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Abstract: Accountability in education has prompted policy makers and practitioners to focus on 
data use for instructional and organizational decision-making. The popular media have seized on 
Value Added (VA) measures as a key type of data use for reforming U.S. schools. Among 
education researchers, however, there are both critics and proponents. We examined data use by 
the district leaders and staff members of 12 schools in a large urban district, with attention to 
the role VA metrics play in their decisions and their conceptions of themselves. VA is only one 
of many types of data that can be used to portray quality. While there was a soft relationship 
between VA and classroom quality measured by CLASS, we found that understanding the 
schools’ contexts, particularly the use of resources and the coherence of actions to improve 
student achievement, greatly enhanced the power of our descriptions. As a result, we suggest 
that policies promoting multidimensional approaches to quality will better capture the 
complexity of education. 
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Resumen: programas de responsabilidad educativa ( accountability) han impulsado el “uso de 
information” (data use) entre responsables de implementar pollticas y educadores profesionales 
tanto para decidir sobre cuestiones de instruction como en decisiones sobre organization. Los 
medios de comunicacion se han enfocado en los modelos de Valor Anadido (MVA) como un tipo 
clave de “uso de information” para reformar las escuelas estadounidenses. Entre los investigadores 
en education, sin embargo, hay tanto criticos y como proponentes de MVAs. Examinamos el “uso 
de information” por directores, profesionales en 12 escuelas en un distrito urbano de gran tamano, 
especialmente el papel que las medidas de MVAs tuvieron en sus decisiones y sus concepciones 
sobre si mismos. MVAs es solo uno de los muchos tipos de “usos de information” que pueden ser 
utilizados para indiicar calidad. Si bien hay una relation moderada entre MVA y calidad de la 
ensenanza medidas por CLASS, encontramos que la comprension de los contextos escolares, 
especialmente el uso de los recursos y la coherencia de las acciones para mejorar el logro de los 
estudiantes, aumento en gran medida el poder de nuestras descripciones. Sugerimos que las pollticas 
que promueven enfoques multidimensionales de calidad cap tan mejor la complejidad de los sistemas 
educativos. 

Palabras clave: modelos de valor agregado; calidad; marco ecologico; uso de information; reforma. 

Ecologias de qualidade educativa. 

Resumo: Os programas de A responsabilidade educacionais {accountability) estao empurrando o "uso 
da informa^ao" (uso de dados) aos responsaveis pela implementa^ao de pollticas e educadores nos 
processos de toma de decisoes sobre questoes educacionais e organizacionais. Os meios de 
comunicafao tem-se centrado em modelos de valor agregado (MVA) como a questao chave no "uso 
de informa^ao" para reformar as escolas americanas. Entre os pesquisadores de educafao, no 
entanto, existem tanto criticos como defensores do MVA. Examinamos o "uso de informa^ao" por 
diretores, profissionais de 12 escolas em um distrito escolar urbano de grande, tamanho 
especialmente o papel que tiverem as medidas de MVA em suas decisSes e suas percep^oes sobre si 
mesmos. MVA e apenas um dos varios tipos de "uso de informa^oes" que pode ser usado para 
indicar qualidade. Embora haja uma relafao moderada entre MVA e qualidade da educa^ao, medida 
por CLASS, encontramos que a compreensao do contexto escolar, especialmente o uso de recursos 
e da coerencia das a^oes para melhorar o desempenho dos alunos, incremento muito o poder de 
nossas describes. Nos sugerimos que as pollticas que promovem a qualidade de enfoques 
multidimensionais de melhor captar a complexidade dos sistemas de ensino. 

Palavras-chave: modelos de valor agregado, qualidade, estmtura ecologica, o uso de informacoes; 
reforma. 


Introduction 

Yes, Virginia, you can have it all. We can get high equity and high student 
achievement. But we will have to go at it differently, very differently (Tucker, 

2012 ). 

Public policymakers have developed what might seem like an insatiable appetite for quality 
indicators in education. Vast amounts of money have been invested in assessments, data warehouses, 
the development of analytical models, and curriculum alignment that should allow the public, 
policymakers and researchers to track the efficacy of schools in promoting achievement. Although 
no one would argue against education quality, and therefore the need to measure it, there is strong 
disagreement about how to enact it in practice, or even what quality means (Elliot, 2011; Zaslow, 
Tout, & Halle, 2011). 
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For some, quality is a matter of investment, identified through expenditures (Heck, 2007; 
Leak & Farkas, 2011). For others, quality is attained through regulatory mechanisms—teacher 
certification, years of experience, or class size, (Darling-Hammond, 1997; Konstantopoulos & 
Chung, 2011). A third group puts their money on the processes of the classroom, focusing on 
teacher-student interaction (Howes & Pianta, 2011; Pianta, Belsky, Houts, & Morrison, 2007). Still 
others frame quality in terms of test scores—the outputs yielded from schooling (Aaronson, Barrow, 
& Sander, 2007; Rivkin, Hanushek, & Kain, 2005). 

Linked to a commitment to accountability, the drive for quality has been pushed by 
methodological advances that translate complex constructs into technical tools (Rockoff, 2004). 
Value added modeling is a highly visible conception of quality, imported from economics and 
designed to identify effects of educational practice (Meyer, 1997). While a prominent feature of the 
quality debate, value added modeling is only one dimension of this complex construct. Building on 
work in early childhood education, we are informed by a multidimensional conceptualization of 
quality, one that highlights the uses for which a measure is designed (Zaslow, Tout, & Halle, 2011): 
The purpose for measuring quality is critical to selecting specific measures. If the 
goal is for overall quality improvement, a broad measure may be most 
appropriate, whereas if the goal is to improve practice in a specific domain, a 
measure focusing in depth on a particular aspect of the environment may be 
more appropriate (p. 393). 

Recognizing that quality is conceptualized differently by different audiences, for different 
purposes, we provide an analysis of quality in context. To do this, we examine how two distinct 
quality metrics—value-added and classroom quality—are promoted in local ecologies and often 
provide contrasting pictures as to why schools achieve success. We interpret these two measures 
of quality within social practices, examining how the conception of a high-, mid-, or low- quality 
school is facilitated and constrained through systems of belief, resources, and value. Specifically, 
we explore the research question: What is the role of local context in conceptions of quality such as value- 
added and classroom quality ? How do these concepts play out in practice? 

Literature Review 

Discussions of quality in education have focused on three aspects of practice with specific 
attention to teacher quality: inputs, outputs, and process. Input approaches specify elements that are 
deemed necessary to set the scene for high quality teaching: per pupil expenditures, class size, or 
teacher credentials, education level and degrees earned. From an input perspective, quality is related 
to investments, and the characteristics of individuals that impact the education process (Pianta, et al., 
2007; Rice, 2003). 

Recently, the outputs of teaching, as measures of effectiveness, have become targets of 
attention from an accountability perspective. Proximal measures of outputs explore student 
achievement/learning, with the expectation of improved outcomes are the result of high-quality 
teaching. For example, student promotion, higher grades, or decreased special education referrals 
could serve as proximal indicators of effective teaching. Distal measures of effectiveness examine 
students’ experiences later in life. Retention in later grades, graduation rates, incarceration rates, and 
adult income have all been used to analyze the long-term effectiveness of educational programs (e.g., 
Chetty, Friedman, & Rockoff, 2011; Reynolds, Temple, Robertson, & Mann, 2002). 

Attainment can be used as both a proximal and a distal measure, as can student growth or 
value-added (VA), given sufficient quality and quantity of data. The attainment perspective takes a 
snapshot of student achievement at a single point in time, as initially required by No Child Left 
Behind (NCLB; e.g., Schwartz, Schmitt, & Lose, 2012). In contrast, examining student growth is 
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designed to assess the contributions, or the VA, of particular schools or teachers (Hanushek & 
Rivkin, 2010; Harris, 2009). Similar to statistical models used in quasi-experimental evaluation 
studies, VA analyses may use existing information that includes all students in a district or state 
across multiple years, allowing population analysis over time (Braun, 2005; Rivers & Sanders, 2002). 
Proponents of value-added models (VAMs) argue that thoughtfully constructed VAMs can account 
for test scale errors, bias in test construction or administration, student background or participation, 
and/or classroom treatments (Ballou, Sanders, & Wright, 2004; Harris, 2011; JYARC: Methodology , 
n.d.). VAM advocates conceptualize their contribution as enhancing our understanding of how 
teachers or schools support student achievement, above and beyond what one would expect from 
non-school factors (Hanushek & Rivkin, 2010). At the teacher level, value-added modeling is 
thought to fine-tune that estimate by parsing out the relative contributions of individual teachers to 
a student’s learning (Chetty, Friedman, & Rockoff, 2011; Rivers & Sanders, 2002). 

The use of VA as an output measure in education has become an area of intense interest in 
education policy and practice over the past three decades, bolstered by the push to examine 
educational practices using economic concepts (e.g., return on investment or cost effectiveness; Yeh 
& Ritter, 2009) and by competition for federal education dollars that emphasize the use of data- 
based decision making (e.g., Race to the Top, n.d.; Spillane, 2012). School leaders began using VAMs 
in the early 1990s when Tennessee launched the Tennessee Value-Added Assessment System 
(TVAAS), a statewide program monitoring teachers’ effectiveness as measured by a value-added 
model (Sanders & Horn, 1998). Over the next two decades, large city school districts (e.g., Boston, 
Dallas, Milwaukee) and entire states (e.g., Florida, North Carolina, Texas) adopted VAMs as ways to 
measure teacher quality (Braun, 2005; Harris, 2011). 

Highly public debates about teacher VA have appeared in The New York Times (Can a Few 
Year’s Data, 2012) and the Los Angeles Times (Grading the Teachers, n.d.). Publication of individual 
teachers’ VA scores in public databases has sparked discussion regarding the purpose and utility of 
these scores among education policy pundits, lawmakers, citizens, and bloggers (e.g. Gates, 2012; 
Wiggins, 2012; Winters, 2012). Skeptics of VAMs argue that we should be cautious in adopting the 
models, particularly if the aim is to use them to make comparisons between teachers or for 
individual personnel decisions because their reliability and validty are suspect (e.g., Au, 2007; 
Corcoran, 2010; Darling-Hammond, Amrein-Beardsely, Haertel, & Rothstein, 2012; Hill, Kapitula, 

& Umland, 2010). There is concern about how VAMs rely on standardized tests, which may not be 
up to the task of individual teacher evaluation (Amrein-Beardsley, 2009; Papay, 2010). There are 
issues of data quality in systems that are only now creating data warehouses capable of tracking data 
appropriately and various kinds of bias make modeling problematic (Briggs & Domingue, 2011; 
Rothstein, 2009). 

Furthermore, there is concern about the uneven track record VA has in education systems. 
For example, results have been inconsistent across models; the same teacher is shown as adding 
value for one model and not adding value in another model (Amrein-Beardsley, 2008; Newton, 
Darling-Hammond, Haertel, & Thomas, 2010; Papay, 2010). Even within the same model, teacher 
VA designations can be volatile, with wide swings from year to year (Amrein-Beardsley & Collins, 
2012; Sass, 2008). It has proven difficult to disentangle teacher’s VA rom student characteristics 
because students are assigned in systematic ways to particular teachers and result in serious 
disincentives for teachers of students at risk (Rothstein, 2009). The highly technical nature of VAMs 
and the proprietary nature of their development have closed off review by anyone who is not an 
econometrician (Amrein-Beardsley, 2009). Finally, there are concerns about the validity of VAMs, 
particularly in using VAMs for making policy or personnel decisions (Amrein-Beardsley & Collins, 
2012; Kupermintz, 2003) 
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Each of these approaches to quality frames it as an individual phenomenon, through a single 
teacher. We suggest a novel approach to quality, one that conceptualizes it from an ecological 
perspective. We have chosen this approach because it makes more complex the concept of 
classroom quality. In the space between quality inputs and outputs, some scholars have argued that 
we should examine the quality of student-teacher interaction because children’s learning and 
development are shaped within social and academic interactions in classrooms. It is through this 
cooperative encounter that instruction from the teacher turns into learning by the student (Stuhlman 
& Pianta, 2009). Through analysis of these classroom interactions, researchers have articulated 
elements of practice that are linked to student behaviors and outcomes (Hamre, Pianta, Mashburn, 

& Downer, 2007; Howes et al., 2008; NICHD, 2004; Wilson, Pianta, & Stuhlman, 2007). These 
elements of practice can form the basis for understanding and evaluating levels of classroom quality 
(LaParo, Pianta, & Stuhlman, 2004). 

This ecological approach draws our attention to the ways that metrics like VA variously 
attend to specific school contexts. By taking an ecological approach we push back on the 
assumption that VA alone can tell policy makers what they need to know about meeting the 
achievement needs of students. While VA accounts for student achievement mainly through 
demographic factors, an ecological approach allows us to consider the influence of context specific 
histories, relationships, and practices. Situating VA in relation to a school’s administrative history 
and practice, the resource allocation investments made by the community, or special funding sources, 
adds depth and breadth to our understanding of the many processes contributing to student 
achievement and classroom quality. 

The ecological perspective initially emerged as a theory of human development. Urie 
Bronfenbrenner (1977) first suggested that human development occurs within nested systems - 
some closer to a child and some further away - but all influencing her individual development. 
Experiences across these nested layers influence who each child is over time. While the ecological 
nested systems theory remains the most popular, Bronfenbrenner’s theory evolved as his own 
research and work developed, and he came to see his nested model as too static. Rather, he argued 
in 1995, development should be considered through process and context: a People Process Context 
Time (PPCT) theory. With PPCT, Bronfenbrenner asserted that meaningful research in 
development had to be grounded both in contexts and in their attending processes. PPCT theory 
provides a rich conceptual model of the kinds of data researchers and policy makers should consider 
when we seek to understand the development of a child, a student, or even a school context (Tudge, 
Mokrova, Hatfield, & Karnik, 2009). 

Students live in multiple systems - the classroom, the school, the community, families and 
home - that interact to influence their development (Bronfenbrenner, 1977). In addition, 
development is also affected by system actors, the temporal context and the processes experienced 
(Bronfenbrenner, 1995). From a policy perspective, with students nested within these ecological 
structures, the various systems, actors and resources that shape their academic development are 
important for consideration. We assert that by examining quality, including VA, within ecologies, we 
are more likely to understand how institutions and communities are resources for development and 
ultimately, student learning and achievement. 

We recognize that the teacher-student interactions that produce achievement come to be 
within particular social, educational, historical, and cultural contexts. These interactions result in 
certain processes of engagement. As such, we argue that an ecological view can provide a more 
nuanced understanding of how individual’s efforts interact to create education quality. This builds 
on the recent work on process and outcome quality, illustrating how school ecologies create 
opportunities and barriers for individual success, locating quality within systems and classrooms (e.g., 
O’Connor & McCartney, 2007). We believe this ecological model has the potential to deepen the 
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understanding VA provides, locating it within exploration of the how and why of student 
achievement. 


Methods 

As part of a larger study of a district’s implementation of a data use reform, we designed a 
mixed methods study that explored how the district worked to enhance the quality of decision 
making by building data-use capacity. This paper comes out of that project, focusing on a sample of 
schools to understand the relations among different metrics of quality within local education 
contexts. This ecologically based approach has been used productively in other studies considering 
practice in local contexts (Graue, Hatch, Rao, & Oen, 2007; Graue, Rauscher, & Sherfinski, 2009). 

The site of this study was a large, Midwestern, urban school district with a long history of 
low student achievement. At the time of this study, the district served over 80,000 students, 84 
percent of whom were minorities (compared to a state average of 24 percent). Nearly 79 percent of 
the district student population was considered economically disadvantaged (compared to a state rate 
of just over 37 percent) and 20 percent had special needs (compared to 14 percent at the state level) 
(State electronic data base, 2012) 

When we began our study, the district had failed to make adequate yearly progress in math 
and reading for the prior six years and had been designated a District Identified For Improvement 
(DIFI) under NCLB. In response, the state department of education issued a demand for corrective 
action requiring the district to make specific reforms to improve student achievement. Our study 
took place within the context of a district-wide data use reform effort that provided professional 
development to district leaders on use of student data for instructional improvement. While data use 
was the impetus for the initial study, this paper focuses on conceptions of quality. 

Calculation of Value-Added Scores 

The VA model used by this district calculates growth in math and reading on the standards 
based state test, controlling for the previous years’ scores and student demographics, and correcting 
for measurement error in year 1 scores. Classroom effects are estimated separately for each grade, 
subject, and year, as are within-school and across-school variation in classroom effects. Univariate 
and multivariate shrinkage estimations are used to obtain more precise estimates, especially for 
classrooms with small class size. Univariate shrinkage assumes that teacher effects are independently 
distributed across the entire district. Multivariate shrinkage assumes that teacher effects are 
correlated within a school. With improved methods of student-teacher links, more precise estimates 
are obtained in later years. 

The district translated VA data into a 2x2 model to present static attainment in relation to 
change over time for the district and for individual schools. Through this 2x2 model, schools were 
classified into four quadrants based upon their attainment and VA (see Figure 1). Where schools fell 
on this quadrant chart became an important designation among staff within the district. While 
schools were not overtly sanctioned due to low VA scores, social pressure was paired with certain 
VA labels. In professional development sessions that we observed, some school staff would tout 
their high VA/high attainment status, while those from low/low schools remained silent. In 
addition, schools were required to document their quadrant placement within their School 
Improvement Plans. 
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We used the quadrant model to select the sample for this project given its importance in the 
district. This allowed us to explore schools that varied in attainment and growth, using their 
quadrant classification as a quality metric. In addition to considering VA/attainment quadrant, we 
took into account contextual elements that shape practice, including a range of school types, from 
suburban ring to inner city, with varying levels of student poverty, English language learners and 
special needs students. Table 1 shows sample demographics in relation to the broader district, 
arranged in order of VA quadrant. 
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Table 1 

School Sample, 2009-2010 


School* 

District 

Mulberry 

Fieldmsh 

Wildwood 

Palmetto 

Nutmeg 

Myrtle 

Grades 


3-8 

PreK- 8 

PreK - 5 

PreK - 5 

PreK-5 

PreK-8 

Total students 

85,376 

403 

533 

217 

520 

371 

331 

VA Quadrant 
Reading 

N/A 

1 

1 

1 

1 

1 

2 

VA Quadrant 

Math 

N/A 

1 

1 

1 

1 

3 

2 

% Prof/Adv 
in Math/Reading** 

49/59 

95/96 

82/72 

94/92 

62/71 

71/68 

37/21 

% Economically 
disadvantaged 

77 

38 

28 

31 

94 

42 

93 

% English proficient 

91 

99 

99 

99 

84 

100 

100 

% African American 

57 

53 

9 

6 

55 

47 

98 

% Latino 

23 

6 

17 

16 

35 

5 

0 

% White 

15 

36 

71 

76 

8 

43 

2 

% Asian 

5 

5 

1 

1 

1 

4 

0 

% Native American 

1 

0 

2 

2 

1 

1 

0 

% with Disability 

19 

4 

13 

7 

23 

23 

24 


*A11 names for sites and persons are pseudonyms. 

** Percentage of students scoring Proficient and Advanced on state test in grades 3-5 or 3-8 in 2008-09 data; Source: 
State school information website. 
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Table 1 (Cont’d.) 

School Sample, 2009-2010 


School* 

District 

Birch 

Acacia 

Elazel 

Sycamore 

Beech 

Hickory 

Grades 


PreK 

- 8 

PreK - 

5 

PreK - 5 

PreK- 5 

PreK-5 

PreK-8 

Total students 

85,376 

459 

247 

368 

305 

353 

325 

VA Quadrant Reading 

N/A 

3 

n/a 

4 

4 

4 

4 

VA Quadrant Math 

N/A 

1 

3 

2 

4 

4 

4 

% Prof/Adv 
in Math/Reading** 

49/59 

82/73 

51/70 

51/47 

61/46 

47/38 

35/19 

% Economically 
disadvantaged 

77 

92 

92 

91 

92 

94 

92 

% English proficient 

91 

84 

43 

100 

100 

100 

100 

% African American 

57 

83 

1 

88 

95 

96 

97 

% Latino 

23 

1 

99 

5 

1 

1 

0 

% White 

15 

3 

0 

5 

3 

2 

3 

% Asian 

5 

13 

0 

1 

1 

1 

0 

% Native American 

1 

0 

0 

0 

0 

0 

0 

% with Disability 

19 

21 

9 

25 

13 

23 

24 


*A11 names for sites and persons are pseudonyms. 

** Percentage of students scoring Proficient and Advanced on state test in grades 3-5 or 3-8 in 2008-09 data; Source: 
State school information website. 

Recording School-level Practices 

Once we had purposively selected our sample schools, the district research office contacted 
the principals, who forwarded our invitation to teachers. At each site, we invited all grade 3-5 
teachers to participate in observations and interviews, with 81 percent participation overall (range 60 
to 100 percent) for a total of 47. Table 2 shows overall participation at each school site. 
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Table 2 


Participants and Participation 


School 

Teachers Invited 

Teachers Participated 

Percentage 

Participation 

Acacia 

4 

3 

75% 

Beech 

4 

3 

75% 

Birch 

5 

4 

80% 

Fieldmsh 

6 

5 

83% 

Hazel 

6 

5 

83% 

Hickory 

4 

3 

75% 

Mulberry 

5 

5 

100% 

Myrtle 

5 

4 

80% 

Nutmeg 

5 

4 

80% 

Palmetto 

5 

3 

60% 

Sycamore 

6 

5 

83% 

Wildwood 

3 

3 

100% 

Total 

58 

47 

81% 


Our school visits lasted two to three days during the spring of 2010. We conducted semi- 
structured interviews with each principal, with volunteers from the learning team (a school level 
leadership group charged with data analysis) and the broader teaching staff. Interview questions 
focused on the school and classroom practices related to data use and individuals’ understandings of 
VA. Questions also worked to link contextual issues in the district and the school to the specific 
practice in classrooms and work to promote student attainment. We provide an example of the 
interview protocol in Appendix A. 

Interviews ranged in length from 40 to 120 minutes and were audio taped and transcribed 
for analysis. We collected relevant documents and artifacts, including worksheets, curriculum guides, 
and school and district reports. We also obtained school-level test score and demographic data from 
the state’s public database. 

Pairs of observers visited the classroom of each participating teacher for two hours 
scheduled to coincide with typical instructional practice. While one observer took descriptive field 
notes that detailed instructional environment and interaction, the other rated classroom quality using 
the Classroom Assessment Scoring System (CLASS; see Pianta, LaParo, & Hamre, 2008). In each 
classroom, the rater observed and coded classroom quality across four twenty-minute cycles of 
instruction with CLASS. These observations generated brief field notes that the observer used to 
assign ratings of 1-7 on each of the CLASS dimensions, referring to detailed mbrics in the CLASS 
manual. Ratings are categorized at three levels: 1-2 = low, 3-5 = midrange, 6-7 = high. Average 
scores were tabulated across four cycles and CLASS dimension and domain scores were calculated 
by both classroom and school. 

Process Quality: The Classroom Assessment Scoring System 

The CLASS provides a common metric for understanding classroom quality, based on the 
relationships between student outcomes and the quality of teacher-child interactions (For more 
information on CLASS, see http: / /www.teachstone.org/research-and-evidence/research-summarv- 
2/1 . As a result of its use for observations in thousands of classrooms, CLASS reliably and validly 
describes the components of classroom interaction that are linked empirically to later student 
outcomes, pre-K through grade twelve (Hamre & Pianta, 2005; LaParo et al., 2004; Pianta et al., 
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2007; Stuhlman & Pianta, 2009). To ensure valid inferences, observers must be trained and tested by 
a CLASS certified trainer so that they reliably use the CLASS framework to rate classroom quality. 
All members of our research team were certified CLASS observers. The constmcts employed by 
CLASS are described in Table 3. Average CLASS scores, and their range, by school are presented in 
Table 4. 

Table 3 


Domains and Dimensions o f CLASS 


Emotional Support 

Classroom Organization 

Instructional Support 

Positive Climate 

The enjoyment and emotional 
connection that teachers have 
with students and the nature of 
peer interactions 

Behavior Management 

How well teachers monitor, 
prevent, and redirect 
behaviors 

Concept Development 
Measures how teachers promote 
higher order thinking and 
problem solving going beyond 
fact and recall activities with 
children 

Negative Climate 

Reflects negativity such as anger, 
hostility, or aggression expressed 
by teachers and/or students in 
the classroom 

Productivity 

Considers how effectively 
teachers manage time and 
create classroom routines that 
maximize learning time in the 
classroom 

Quality of Feedback 
Considers how teachers extend 
student learning through their 
responses and participation in 
activities 

Regard for Student Perspectives 

Captures the degree to which 
teachers’ interactions with 
students and classroom activities 
place an emphasis on students’ 
interests, motivations, and points 
of view 

Instructional Teaming Formats 

Focuses on how teachers 
engage students in activities 
and facilitate activities so that 
student learning is maximized 

Language Modeling 

Reflects the extent to which 
teachers facilitate and encourage 
student language 

Teacher Sensitivity 

Reflects teachers’ responsivity to 
individual students’ academic and 
emotional needs 
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Table 4 


CLASS Scores and Range by School 


School 

Emotional 

Support 

Classroom 

Organization 

Instructional 

Support 

School 

CLASS 

Total 

Mulberry Average Range 

5.76 

5.5 to 6.0 

5.92 

5.42 to 6.42 

4.63 

3.50 to 5.63 

5.48 

5.25 to 5.80 

Fieldrush Average Range 

5.59 

4.69 to 6.44 

5.18 

3.75 to 5.92 

3.88 

2.69 to 5.13 

4.93 

3.75 to 5.97 

Wildwood Average Range 

5.19 

4.75 to 5.94 

5.22 

4.50 to 5.92 

3.78 

2.83 to 4.44 

4.84 

4.10 to 5.29 

Palmetto Average Range 

4.92 

4.38 to 5.63 

5.36 

4.58 to 6.17 

4.20 

3.06 to 5.17 

4.86 

4.19 to 5.65 

Nutmeg Average Range 

5.40 

4.81 to 5.94 

5.25 

4.42 to 6.42 

3.80 

3.08 to 4.88 

4.90 

4.18 to 5.87 

Myrtle Average Range 

5.28 

4.38 to 6.13 

5.00 

4.67 to 5.50 

3.60 

2.25 to 5.56 

4.71 

3.95 to 5.86 

Birch Average Range 

4.88 

4.31 to 5.38 

5.48 

4.67 to 6.33 

3.36 

2.75 to 4.06 

4.75 

4.03 to 5.11 

Acacia Average Range 

5.56 

5.38 to 5.94 

5.67 

5.50 to 5.75 

3.31 

2.88 to 3.92 

4.94 

4.75 to 5.05 

Hazel Average Range 

4.85 

4.06 to 5.69 

4.75 

3.33 to 6.25 

2.85 

2.33 to 3.75 

4.31 

3.53 to 5.52 

Sycamore Average Range 

5.61 

5.19 to 5.94 

5.17 

4.33 to 5.75 

3.50 

2.75 to 4.25 

4.83 

4.38 to 5.20 

Beech Average Range 

4.13 

3.63 to 4.50 

4.46 

2.75 to 6.17 

2.90 

2.0 to 3.69 

4.02 

2.88 to 5.04 

Hickory Average Range 

3.60 

2.44 to 4.94 

3.78 

2.58 to 4.83 

2.48 

1.5 to 3.69 

3.31 

2.51 to 4.61 


Data Analysis 

Our analysis brought together teacher and school VA scores, CLASS ratings, notes from 
classroom observations, transcripts of interviews with school staff, and school documents. The 
specific work for this paper began with the examination of the CLASS ratings of quality in relation 
to school VA. Based on the assumption that classroom quality comes to be in the context of school 
ecologies, we explored the relationship between output and process quality. We conducted a linear 
correlation analysis, calculating the Pearson correlation coefficient. We used the following variables 
in our analysis: school-wide three-year n weighted VA scores in math and reading, average CLASS 
ratings across sampled classrooms at both the domain and aggregate levels, and the percent of 
students eligible for free and reduced lunch. These metrics were chosen as different aspects of 
quality, and we included socioeconomic status given its relationship to resources in school contexts 
and student achievement. 

This approach had both inductive and deductive elements. We observed that the variation 
within schools could be as great as the variation between schools. It was therefore important to 
closely examine patterns in administrative practice, teacher beliefs, types of classroom interaction, 
and school attainment. In contrast, we also worked from the ratings to the fieldnotes and interviews, 
exploring likenesses and differences in high, midrange, and low rated classrooms. For each school, 
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our field notes were used as a crosscheck for CLASS ratings. We read the CLASS ratings in relation 
to the full fieldnotes, working to illustrate the ratings in practice. We looked for confirming and 
disconfirming evidence for the patterns we identified. Supported by the qualitative data analysis 
program NVivo, the research team analyzed data sources using a shared set of codes. These codes 
reflected the assumptions that guided the research as well as concepts that were new to the project. 
We shared memos (Graue & Walsh, 1998; Maxwell, 1996) that detailed analytical conceptualizations 
that linked coding categories and illustrated crosscutting themes (Saldana, 2009). 

With a mixed methods approach it is important to be clear about how we made research 
inferences. We took a strong interpretive approach (Graue et al, 2007; Graue et al., 2009), with 
attention to how the local school context set the stage for both school and classroom quality. We 
relied on the notions of trustworthiness and transferability, where the readers made connections 
from the case to their own knowledge and experience, evaluated by the degree to which the findings 
appeared to resonate with the reader from their reading of the existing literature and the 
presentation of the case relative to the literature (Lincoln & Guba, 1985). 

Given our interpretive approach, we are clear about the limitations of such a study. As a 
strength, we carefully assembled a bricolage of data sources related to education quality and 
examined them in relation to one another and in specific local contexts. We recognize that our 
sampling techniques limited the inferences made from our interpretations; the results presented are 
not population inferences. Instead we conceptualize them as quality in context, presenting carefully 
considered examples to enrich our thinking about quality. 

Results 

Our analyses focused on two levels. The first level examined the relationship between 
outcome and process measures of quality within our 12-school sample at both the school and 
classroom levels. This analysis is presented through correlations of the quality measures (CLASS and 
VA) in relation to the percentage of students in poverty in each school. The second level described 
four schools that exemplified how school ecologies (i.e., their histories, community resources, and 
administrative supports) provide a context for better understanding “quality.” 

Correlation of VA and CLASS for Schools and Teachers 

Given our interpretive approach, our attention to statistical significance is slightly different 
than if we were arguing for strong population inferences. With the small number of schools and 
teachers, we lack statistical power, but more importantly we see the correlation coefficients as rough 
indicators of shared variation, analogous to a scatterplot. We consider it in relation to the case study 
data to analyze how these two measures of quality are related in these specific contexts. We will note 
however that with the small sample the statistical significance indicated in the tables could be 
thought of as more pattern than noise. 

At the school level, these two measures of quality, VA and CLASS, were positively 
correlated (see Table 5). The CLASS Instructional Support dimension, which describes how teachers 
extend student thinking, provide feedback and support language, is most strongly related to VA, 
both in a single year and with a three-year average. This is in keeping with other research that has 
linked instmctional support to school outcomes (Hamre & Pianta, 2005; LaParo et al, 2004; Pianta 
et al, 2007). Both VA and CLASS, as measures of outcome and process quality, are negatively related 
to the density of students living in poverty in a school (see Table 5). This, too, is congment with 
previous research that finds that students in low-income schools receive lower quality instruction 
and as a result, have lower achievement (Adamson & Darling-Hammond, 2012; Burchinal et al., 
2011 ). 
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Table 5 

Correlation Between School 1 AA and CLASS 



VA/Rdg 

1 year 

VA/Math 1 
year 

VA/Rdg 3 
year average 

VA/Math 3 
year average 

% of Students 
Economically 
Disadvantaged 

Emotional Support 

.42 

.32 

.53 

.38 

- .48 

Classroom 

Organization 

.54 

.41 

.51 

.58* 

- .36 

Instructional 

Support 

.65* 

.63* 

.71* 

.64* 

- .59* 

CLASS Total 

.55 

.43 

.58 

.56 

- .50 

% of Students 

Economically 

Disadvantaged 

-0.81** 

- .53 

- .73* 

- .35 

— 


* p < .10; **p <.05 


At the level of an individual classroom, the relation between classroom quality (CLASS) and 
teacher quality (VA) was weaker than at the school level (see Table 6). Statistically this makes 
sense—more data points reduce the error—but from a practical point of view, this finding is 
problematic. If VA and CLASS are both measures of quality, and if policy decisions, like teacher 
evaluation, hinge on these measures, they should be similar rather than different to be valid 
indicators. A focus on VA alone could miss important information about education quality. We 
argue, therefore, that policy makers should consider, as we do in the next section, what other factors 
may be influencing classroom quality. 


Table 6 

Correlation Between Classroom VA and CLASS 



Math VA 

Reading VA 

Math VA 

Reading VA 


1 year 

1 year 

3 year 

3 year 

Emotional Support 
(E.S.) 

.23 

.22 

.25* 


Classroom 

.19 

.24 

.11 

.37* 

Organization (C.O.) 

Instructional Support 
(I.S.) 

.28* 

.29* 

.20 

40*** 

CLASS Total 

.22 

.23 

.14 

4Q*** 


*p < .10; **p < .05; ***p <.001 








Ecologies of education quality 


15 


Broadening the Focus Ecologically 

While VA pays some attention to context through its inclusion of student demographic data, 
it is a statistical consideration, limited to what can be portrayed in the model. We wondered how a 
more nuanced view of context might help us understand how quality comes to be locally and, 
therefore, how VA reflects or obscures the contribution of local ecology to schooling. We turned to 
the development of descriptive portrayals of the schools, relying on documents, observations, and 
interviews, to provide a window of the local context, and to better understand how this related to 
the quantitative data we had gathered. 

The kind of description that would provide insight into context would have been unwieldy 
with all twelve schools in our sample, so we have narrowed our descriptive focus to four sites. We 
describe four schools that represent different VA quadrants, profiles, and communities to explore 
how school VA translates into particular school practices and further, how they map onto individual 
teacher measures of quality. Table 7 provides a longitudinal look at school VA quadrant placements 
over an 8-year period. Mulberry consistently occupies Quadrant 1 and Hickory predominantly 
Quadrant 4. Sycamore more often sits in Quadrant 2, while Nutmeg goes back and forth between 
Quadrants 1 and 3. 

Table 7 


1 YA. Quadrants Over Time for Four Sample Schools 


School 


2001-02 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

Mulberry 

Reading 

1 

1 

1 

1 

1 

1 

1 

1 

School 

Math 

1 

1 

1 

1 

1 

1 

1 

1 

Nutmeg 

Reading 

3 

1 

3 

1 

1 

1 

1 

3 

School 

Math 

3 

1 

3 

1 

1 

3 

1 

3 

Sycamore 

Reading 

1 

1 

3 

2 

2 

2 

4 

2 

School 

Math 

1 

1 

3 

4 

4 

4 

4 

2 

Hickory 

Reading 

4 

2 

4 

4 

4 

2 

4 

4 

School 

Math 

2 

4 

4 

4 

4 

2 

4 

4 


In the following sections we describe each school context and the ecological qualities that 
shape students’ and teachers’ experiences. As an overview, we illustrate the CLASS-VA relationship 
at the four schools at the individual teacher level. If classroom quality (measured by CLASS) and 
teacher quality (measured by VA) represented the same construct, we would expect a positive linear 
display. This graph, which represents individual teachers’ scores, is less linear than might be 
expected for measures that are used for high stakes decisions. 
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Figure 2. CLASS Instructional Support and VA by Teacher 
The Contexts for Quality 

Mulberry: Overflowing assets 

A high VA/high attainment school (Quadrant 1), Mulberry was overflowing with assets. The 
students at this selective Grade 3-8 school for the talented and gifted were the highest performing in 
the district. The school served few students with disabilities and had a higher proportion of white 
students (36 percent) than the district overall (15 percent). 


Table 8 

School, CLASS, and 1 FA Data for Mulberry School 


School 

CLASS Scores 


Emotional Support 
5.8 


Classroom Organization 
5.9 


Instructional Support 
4.6 


Key % eligible FRL % Prof/Adv in Math % Prof/Adv in Reading 

Demographics 38 95 96 


VA Quadrant 
2008-09 


Math: 1 


Reading: 1 


Mulberry families were more affluent than their district peers, and the language of economic 
capital came up when a teacher described how parents saw the school: 

I think that parents consider this school an investment . . . these parents are 
investing so much time and effort, and they know that it will pay off in the end. 

And their investment and privilege is taken seriously [Mulberry teacher]. 

A sense of exceptionalism pervaded every aspect of life at Mulberry. The enriched 
educational experience at Mulberry, beyond the means of most district elementary schools, was 
possible because many Mulberry families could contribute (both financially and with their time) 
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and because teachers worked with the students to raise funds. The school’s website showcased 
its assets, programs and events by providing links to media coverage of the school and actively 
soliciting donations well beyond what the typical bake sale would bring in. One page suggested 
donations ranging from $100, to secure a bus for a field trip, to $40,000, to fund a full-time 
teacher of the arts. The physical plant was also a treasure. One of the oldest school buildings in 
the district, it had been lovingly restored and maintained. The classrooms were beautifully 
equipped with a variety of high tech teaching tools and teachers were well-trained in their use. 

In a district that stmggled to provide any instruction outside the classroom core, education 
at Mulberry was multidimensional. Supplementing the art and music specialists from the district, 
Mulberry raised funds to hire a theater teacher and all students took tap dancing from a dance 
instmetor. Not content to bring the world into the school, teachers took students on a variety of 
learning trips to destinations ranging from the state capital to Europe. According to one teacher, 
Mulberry was a place that had “all of those wonderful programs...that complete the whole child.” 

The teaching staff and their instructional expertise were assets as well. A dedicated and 
talented teaching staff viewed themselves as educators who pushed their students to reach their 
potential. Five world languages were taught and teachers could draw on a vast network of parent 
volunteers to help with almost any task and it was expected that all parents followed through on 
school work at home. 

Classroom quality was high at Mulberry, with uniformly strong CLASS ratings mirroring the 
positive VA ratings. Mr. Minsky, a 4th grade teacher, exemplified the wealth of teaching expertise at 
Mulberry. In sessions that we observed, Mr. Minsky drew on the considerable resources available in 
his classroom to build his students’ knowledge. For example, he activated students’ unique 
knowledge (referring to a trip that a student took), and worked to clarify their learning by making 
connections (referring to earlier books they had read or discussions they had had). 

It would be easy to say that Mulberry is a model school that could be held up as an example 
for reform. If the ecological context of Mulberry could be replicated across this district, it is a good 
bet that achievement would go up, teacher satisfaction would improve, and children’s futures would 
not seem so bleak. But was it the instructional context that added value to these students’ education? 
Or was the narrative more complex, an example of Floliday & Flerzog’s lyrics: “Them that’s got shall 
have. Them that’s not, shall lose...” (1939). What made education work so well at Mulberry was a 
combination of the school’s tremendous resources and how those resources were used in a coherent 
way by staff working together toward the same goal. They had little resistance to work against and 
every advantage working for them. With parents supporting and contributing to their efforts, the 
students and staff at Mulberry enjoyed an embarrassment of riches. Given every advantage, it is not 
surprising that this school has had high VA scores year after year. Assuming that other schools 
could somehow follow the lead of Mulberry through instructional reform, however, ignores the very 
different circumstances in which other schools in this district exist. 

Hickory: A school all tattered and torn 

Unlike Mulberry, Hickory was not a school with either a high VA or a high attainment score, 
nor did they have overflowing assets. Instead, Hickory was a more common example of a school in 
this district: struggling under the weight of the burdens carried by its students and families. The 
school served a predominantly African American and low-income population and is nested deep 
within one of the poorest zip codes in the United States. 



Education Policy Analysis Archives Vol. 21 No. 8 SPECIAL ISSUE 


18 


Table 9 

School, CLASS, and HA Data for Hickory School 


School 

CLASS Scores 


Emotional Support 
3.6 


Classroom Organization 
3.8 


Instructional Support 
2.5 


Key % eligible FRL % Prof/Adv in Math % Prof/Adv in Reading 

Demographics 92 35 19 


VA Quadrant 
2008-09 


Math: 4 


Reading: 4 


Hickory stood as a stark contrast to Mulberry, with some of the lowest CLASS scores from 
our sample (especially in Instmctional Support) and low VA/low attainment scores. It was a school 
on the brink. Whereas the entrance to Mulberry was stately, the entrance to Hickory was a large, 
plain, locked black double door with a buzzer. Inside, the barren green walls and dusty floors gave 
the building an institutional feel. While there were provisions for security, they did not seem to be 
used; the security guard’s desk was unoccupied. 

What the worn public face of Hickory did not show was that the school received 
supplementary materials due to the extreme poverty of its students and their failing test scores. For 
example, the school received funding for staff to provide supplementary services to students. 
However, each additional staff member represented a different initiative at work in the school: a 
math teacher for the math grant; a reading coach for the literacy initiative. As each initiative was 
uncoordinated with others, they seemed to take on a hit or miss approach. These adults walked the 
halls, pulling their students from classrooms for supplemental instruction, regardless of the desires 
of the classroom teacher. These supplementary staff and the classroom teachers had little to do with 
each other beyond the fact that they served the same students. According to classroom teachers, 
they failed to provide cohesive support for the students that needed it most. 

Whereas Mulberry often successfully relied on families to raise money for technology. 
Hickory received grants for Smart boards in each classroom and for a central computer lab to 
support student learning. However, due to a lack of funds to train and support teachers or provide a 
teacher for the computer lab, many of these new tools were not integrated into instruction. During 
an observation of a classroom teacher at Hickory, one of our researchers noticed a Smart board that 
was entirely hidden behind stacks of old books. When she later asked the classroom teacher about 
the Smart board, she was told that the teacher had never been given support in using the board, so 
she couldn’t use it. Apparently this was the case in many classrooms through the school. 

Compared to Mulberry teachers, who only got stronger with strategic and concerted use of 
their resources, Hickory staff was too tired or overwhelmed to take up new opportunities and 
materials directed to their school. According to many of the teachers that we observed, the school 
experienced a hit and mn approach to goods and services: a grant or organization would allocate 
resources at Hickory and then move on to another school. Hickory staff was left to contend with 
these contributions without training or support. The inability of Hickory’s teachers’ to activate and 
utilize these resources, and be supported in their use, minimized their benefit. This may have hurt 
schools like Hickory in another way as well. Being the beneficiaries of material goods, but not 
ongoing support, schools like Hickory ran the risk of looking ungrateful and unmotivated to use the 
goods that they were given. 

These patterns were visible across a variety of data. The CLASS scores at Hickory were the 
lowest in our sample and illustrated a typical image of a low VA/low attainment school. An 
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instructional example of the lack of coordination of practice at Hickory was seen in Ms. Connor’s 
fifth grade classroom. The beginning of the day was beset with interruptions. Teaching started and 
stopped around PA announcements or phone calls from the office; more than 10 of her students 
were pulled out for various services. The frustration on Ms. Connor’s face quickly became apparent 
as she had to repeatedly retrace and repeat the content she was reviewing. 

While the pull out system allowed students to get remedial and intervention services, it put 
those students behind what was happening in their classrooms. What might have been a well- 
designed lesson became a disjointed instructional experience for these students. Even students who 
were not pulled out were affected, as the forward motion of the lesson slowed to a crawl each time a 
specialist came in to either pull or return a student. With this system in play, intended learning was 
dismpted by many intermptions. 

The Hickory community worked in a challenging context—a mndown facility, children in 
poverty, a history of low performance. These issues had brought supplementary resources from 
various groups, but the inability to coordinate the structure and assets of the school sustained the 
problems that limited practices at Hickory. What the principal said was often in conflict with what 
the Learning Team and teachers shared. Likewise, the school leaders and teachers maintained a 
narrow view of their duties, only focusing on their own responsibilities. The principal, for his part, 
was primarily concerned with his administrative role - what happened in the classrooms was the 
purview of the teachers. When asked to reflect on their practices, both the principal and the 
Learning Team pointed fingers at other—the principal believed the Learning Team needed to be 
more focused, whereas the Learning Team thought the teachers needed to be better qualified and 
the principal more hands on in actualizing resources. Lastly, the teachers blamed the students’ home 
lives for their poor achievement levels. Given their focus on passing the blame, the staff could not 
leverage the resources available to develop a coherent plan to improve achievement. 

Nutmeg: Running hard to stay in place 

Unlike Hickory, Nutmeg Elementary School appeared to have many advantages. Bordering 
an affluent suburb, the neighborhood was predominantly white and middle class, largely populated 
by teachers, firefighters and police who are required to live within the city limits. The school was 
surrounded by a sizeable lawn and large areas dedicated to student play. The school building was a 
rambling, 1960’s open concept school. The long hallways were clean, and the walls a riot of color 
with posters, bulletin boards and student work. Outside each room was a small white board with the 
day’s plan in terms of concepts, tests, curriculum, and technology. Despite these outwards markers 
of a school primed to provide its students with advantages. Nutmeg was in fact a low VA/high 
attainment school (see Table 10). In addition, the range of classroom quality was wide at Nutmeg, 
with some teachers scoring very high on the CLASS and others in the mediocre range. 

Table 10 

School, CLASS, and VA Data for Nutmeg School 

School Emotional Support Classroom Organization Instmctional Support 

CLASS Scores 5.4 5.3 3.8 

Key % eligible FRL % Prof/Adv in Math % Prof/Adv in Reading 

Demographics 42 71 68 


VA Quadrant 
2008-09 


Math: 3 


Reading: 3 
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One of Nutmeg’s main assets was the student population, which was somewhat similar to 
that of high performing Mulberry School. The two schools had about the same proportion of low- 
income students, both well below the district average. The percentages of students of each race were 
about the same as well. In addition, the school had a resourceful, high energy principal who gave 
clear instructions (such as requiring the teachers to write their day’s plan on the whiteboard outside 
the classroom doors) and walked the halls, checking in with students and teachers to make sure that 
these instructions were being carried out. 

However, unlike Mulberry, which pulled students from across the city, Nutmeg was 
primarily a neighborhood school. In fact, as a magnet school for gifted and talented, Mulberry pulled 
the best students away from schools like Nutmeg. The principal, Mr. Reagan, expressed frustration 
at how the magnet schools skimmed off “his” students after they finished second grade: 

I've been here five years now—and I know that from grades 3-5 I've probably lost a 
good 20-25 students over the course of the time. Those are my proficient and 
advanced students. So I could just imagine what my [state test] and Benchmark 
scores would look like had those third, fourth, and fifth graders stayed here [Nutmeg 
Principal interview]. 

Losing 4-5 students per year was clearly painful for the principal, although it amounted to just 1 
percent of his students annually. However, for the principal, this was a main reason that their 
school was not shown to be adding value like the District wanted. The loss of students to more 
“prestigious” schools was a professional affront for a principal who felt that his school had 
much to offer and that his achievement scores were suffering as a result. As we analyzed the 
practices at Nutmeg, we realized that Mr. Reagan was primarily focused on the school’s image to 
the outside world. 

At the school level, the strong procedural leadership of the principal gave the school an 
outward direction. Mr. Reagan gave many examples of how the school staff had been ahead of 
the curve on the most recent district reform initiatives because of their participation in nearly 
every pilot opportunity that the district offered. While much of Mr. Reagan’s focus was on 
managing the outward perceptions of his school, he was also in tune with his staff in terms of 
their expectations of him. In answer to our question, “What would you like the principal to 
know...?” nearly all of the teachers answered, “Oh, he knows.” As one teacher told us: 

He's very supportive of whatever we need. Whatever I need, he's pretty much willing 
to provide it, whether it's assistance in a discipline situation or—he's just aware. He's 
involved; it's not like he's hiding out in the office, and he's aware of what students 
are in the building, and he deals with the issues [5th grade teacher, Focus Group]. 

Mr. Reagan was protective of the teaching staff, questioning the logic of our research project 
and conveying teachers’ concerns to us before we entered classrooms. He was a school leader, 
but not an instructional leader. 

Nutmeg staff saw themselves as the hardworking but under-appreciated stepchild. As a low 
VA/high attainment school, staff were concerned that their identity as high achievers was 
questioned because they were not adding value. With good test scores and strong data practices, it 
was irritating that they were not recognized as an outstanding school. That mismatch made it easy to 
focus on the attainment scores as the favored way they identified themselves. It was additionally 
irritating, noted school staff, that lower achieving schools, such as Hickory, were the recipients of 
resources that these teacher felt they could use to improve their instruction. In addition to losing 
students to other schools, teachers noted that Nutmeg would be losing its class size reduction 
funding for the coming year, so K-3 class size would rise, and the librarian would be forced to return 
to the classroom. Special classes were being eliminated, and along with them, coordinated planning 
time. 
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As a group. Nutmeg teachers were clearly suffering from “reform fatigue.” Their principal, 
eager to capture the attention to District level staff, signed the school up for every reform being 
piloted, and staff, while impressed with Mr. Reagan’s confidence in their abilities, sometimes had 
trouble choosing which reform to pay attention to: 

If literacy centers are going to be it, they need to say that's going to be our focus 
for next year and provide in-servicing and training over the summer, where you 
can hit the ground running, and do it. And that's the focus this year, and not keep 
changing what's going on. I mean, it's such a reactive district now: anything 
comes out now, it's, “Okay, we've got to change; we've got to do something 
else.” And they have to throw something else out. It seems like it's just so 
reactive that nothing really stays in place and is given a chance to show whether 
it's good or bad [5th grade teacher. Focus Group]. 

The changing outside reforms and district agenda were indicative of what drove the education 
engine at Nutmeg—the perspectives of outsiders. Overall, the pressure placed on Nutmeg to 
reach standards had redirected attention from the internal—what the staff intuitively knew they 
needed to be doing to improve their school, to the external—what District administrators, and 
Mr. Reagan, were telling them they should be doing. 

Sycamore: A.U oars in the water but paddling upstream 

A sprawling mid-century style school building, Sycamore was located in a predominantly 
working class, African American neighborhood. The entrance to the school was light and airy with 
an area devoted to family materials - from upcoming neighborhood events to resources for free 
healthcare and dental services for children. At the beginning of the school day parents and children 
moved quickly down hallways and into the gym, which doubled as the cafeteria and was teeming 
with children eating breakfast. 

While Sycamore seemed to be a low VA/low attainment school like Hickory, serving an 
overwhelmingly low income, African American population, the schools could not have been more 
different. Unlike Hickory, Sycamore had some of the highest CLASS scores in our sample (see Table 
11 ). 


Table 11 

School, CLASS, and UH Data for Sycamore School _ 

School Emotional Support Classroom Organization 

CLASS Scores 5.6 5.2 


Instructional Support 
3.5 


Key % eligible FRL % Prof/Adv in Math % Prof/Adv in Reading 

Demographics 92 51 47 


VA Quadrant 
2008-09 


Math: 2 


Reading: 2 


The principal was absent the morning of our visit, away at an administrative meeting, but 
had prepared her staff in advance for our visit. In her stead, the Curriculum Generalist (an 
administrative team member) maintained an easy sense of order, using a walkie-talkie, and carrying a 
clipboard. Mothers with infants and toddlers sat at tables with their school-age children and talked 
to them as they ate breakfast in the cafeteria before school started. Across from the cafeteria, the 
office was open and buzzing. The secretaries laughed with parents who dropped off notes or let 
them know of a child needing to leave early. The energy level in the school was high, happy, and 
contagious. 
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This entrance scene paints a picture of the overwhelmingly positive and family-oriented 
spirit of Sycamore. From this description, or its proximity to a largely white and affluent inner-ring 
suburb, one might think that this was a well-resourced, suburban school. Instead, Sycamore 
stmggled daily to meet its students’ academic needs. Students were bussed to Sycamore from across 
the district, most coming from neighborhoods within the city that are among the poorest in the 
nation. 

Consistently moving back and forth between being high VA/low attainment (Quadrant 2) 
and low VA/low attainment (Quadrant 4), Sycamore was labeled a success one year and a failure the 
next (see Table 7). Though achievement was elusive, the mission of the school - to serve children 
and their families - could be seen in the friendly interactions among families and staff and the 
instructional practices aimed at high achievement. Sycamore staff spoke with one voice about the 
goals of the school - they wanted to do everything within their power to make their students 
successful. 

The coherent nature of the school staff, mission and goals was at the core of everything we 
saw at Sycamore. Each interview portrayed an image of a healthy school culture united in a common 
mission of student success. Interestingly, gains in one area — either math or literacy — were often not 
sustained when gains were made in the other. It became clear that as attention shifted to areas of 
need, teachers were able to promote gains in one area but that this often lead to losses in another. 
These trade-offs in VA scores point to the largest roadblock to success at Sycamore: an overall lack 
of resources. 

Not doing poorly enough to garner extra resources and also not doing well enough to 
overcome obstacles to student success. Sycamore sat on a bubble in the district. In some ways, more 
failing students would have secured them greater support. But this ran contrary to the strong school 
mission and goal of achievement for all students that was at the heart of Sycamore School. Because 
resources were limited at Sycamore, chart paper and chalkboards were the state of the art classroom 
technology, class sizes were larger than the other schools in our sample, and no specials were 
available to students. 

In general, school staff seemed to be paddling against a current that daily threatened to 
sweep their students away. It didn’t matter that they all had their oars in and were rowing together - 
they were still using a rowboat when to make headway they needed a motor. While teachers actively 
channeled their limited resources towards academic achievement within a context of developing 
respectful and ethical citizens, it wasn’t enough. For Sycamore, the disconnect between the 
overwhelming unity of the staff and mission and the stark reality of a lack of resources kept them 
from accomplishing the goals necessary to add enough value to raise their students’ academic 
achievement. 

Several characteristics of Sycamore were unique. At this year-round school, the rhythm of 
the school was different from the others in our sample. The school’s distributed leadership style 
valued the professional knowledge and abilities of staff and its active engagement of families. More 
than any other school in our sample. Sycamore had a large number of African American teachers 
and staff, many in leadership positions. Families were omnipresent at Sycamore. Families could be 
seen in every part of the school, in the cafeteria and in classrooms, allowing mothers who worked 
the third shift to see their children during the school day. Teachers seemed to view these visits as a 
matter of course and an opportunity to greet and chat with the parents, rather than an intrusion on 
their instructional time. These positive attributes met structural forces in resource allocation that 
limited the ability to translate these assets into achievement. 

At Sycamore, we found an interesting example of how classroom quality can be disturbed in 
a school united in its purpose, but fragile in its delivery. On the morning we observed Ms. Driver’s 
third grade class, we found an inviting, spacious classroom that hummed with activity at the start of 
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the day. The intellectual quality gained momentum as the morning proceeded and the children 
worked in literature groups interpreting complex texts. Whole group instruction went smoothly with 
this group of students and the level of discussion was high. At 10:00 AM, Ms. Driver was called to 
an IEP meeting. A paraprofessional took her place, continuing the literature lesson. The group 
immediately began to unravel and the lesson fell apart. Children began to walk around the classroom, 
and we saw a human version of herding cats. Ms. Driver returned after 25 minutes; the children 
immediately coalesced, regaining focus and interest in their work. Table 12 illustrates how support 
for learning broke down when the teacher was pulled for other duties. Observation cycles involved 
twenty minutes of observation followed by ten minutes of rating. 

Table 12 


CLASS Scores for Teacher and her Paraprofessional Substitute 


Teacher for each 20 
minute scoring 
segment 

Emotional 

Support 

Classroom 

Organization 

Instructional 

Support 

Total 

Ms. Driver 

4.40 

5.33 

2.33 

4.40 

Ms. Driver 

5.80 

5.67 

6.00 

5.80 

Paraprofessional 

3.20 

2.00 

2.00 

3.20 

Ms. Driver 

5.80 

6.00 

5.67 

5.80 


Sycamore provides a window to the challenges of systemic reform in an impoverished 
context. In many ways, the staff had embraced the idea of whole school reform. They had a 
coherence of message and a collaborative spirit that allowed them to work together toward the goal 
of improving achievement. But given the system of rewards and punishments in current education 
policy, their school still struggled with pulling together the resources needed to move forward. In 
addition, Sycamore’s VA scores, more consistently low than not, obscured the school’s potential. 
Behind the VA scores we saw shifting attainment, with momentary success in supporting 
achievement until the focus of effort was moved to something else. Though both Hickory and 
Sycamore had low VA scores, an ecological perspective gave us a window on important differences 
in how staff at these two schools negotiated unique challenges. 

Discussion 

It should be no surprise that policy makers have chosen to gloss over the long, 
difficult endeavor to identify the elements of effective teaching, and move 
straight to an emphasis on effective teachers leaving what makes some teachers 
more effective than others as an unsolved mystery (Gamoran, 2011, p. 202). 

The current focus on quality in education, nurtured in a context of accountability, has 
sharpened attention to the measurement of teaching practice and its outcomes (Cuban, 2007). In this 
context we undertook a study to examine different metrics of quality with attention to local 
enactment. We were convinced that situating quality in local sites of teaching and learning would 
help us illuminate how context mediates the process and outcomes of instruction. 

We found two elements important for understanding local practice. The first, resources, is 
often seen as the Holy Grail of education (Cohen, Raudenbush, & Ball, 2003). Resources may be 
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intellectual, material, environmental, or social assets that can be used in education. The second 
element, coherence , describes the consistency of goals, purposes, programs, and staffing (Madda, 
Halverson, & Gomez, 2007). Our analysis indicated that there is not a linear relationship among 
quality and neither resources nor coherence. Mulberry, a school with the highest quality ratings on 
either of our two measures, had abundant resources across cultural, social, and economic capital and 
strong coherence. That, however, was the only simple linear relationship that we could find. When 
we considered resources input by the district, the lowest rated schools also had more resources 
poured into them than those whose quality was low overall, but often drifted towards the middle. 
While this might seem appropriate from an equity perspective on resource allocation, we found 
Cohen and colleagues’ assertion quite compelling: 

The instructional effects of conventional resources depend on their usability, 
their use by the agents of instruction, and the environments in which they work. 

When added conventional resources appear to directly affect learning, it is 
because they are usable, because teachers and students know how to use them, 
and because environments enabled or did not impede their use...If these ideas are 
correct, then when added resources lie outside the range of teachers’ and 
students’ knowledge, norms, and incentives, they will have no discernable effect 
(Cohen et al., 2003, p. 132 ). 

This was borne out in our analysis. In schools like Mulberry, resources were available and 
activated along every dimension imaginable. In a district with crumbling buildings and perennial 
struggles to find funding for technology, programs and staff. Mulberry was a well-equipped 
education palace. Mulberry’s identity as a school for the talented and gifted framed every aspect 
of work and provided a unifying theme for practice. Teachers, students, and families enriched an 
educational environment that included supplemental trips and activities, and highly engaging 
pedagogy. 

In contrast, schools like Hickory were housed in lifeless buildings with a sense of despair. 
Serving some of the most vulnerable students in the city, these schools were more densely staffed 
and resourced, but the assets were only marginally related to instmction. With low profile leadership 
and without a mission, the staff found it difficult to use the resources they had. The dusty Smart 
board hidden behind piles of books is a metaphor for this school’s struggles - it was an investment 
in a school in need but not enough of an investment to allow it to be used. The main work of 
Hickory staff was to avoid losing the education game, keeping students in the building and out of 
trouble. 

Schools like Nutmeg or Sycamore had mixed resources and coherence. Nutmeg was a well- 
run school with a history of data use and neighborhood resources. The principal was focused on 
external audiences and less on his role as instructional leader. As a result, staff invested much energy 
and attention in communication of standards and activities, but there was less coherence in how or 
why they were doing what they were doing. Sycamore had the opposite issue. The school had a clear 
mission, and all teachers and staff spoke from the same script in terms of their goals for their 
students and the ways they worked. Unfortunately, they were not failing enough to garner the 
additional resources given to schools like Hickory. So they stmggled to stay up to date in their 
practice, relying on time proven tools like chart paper and bulletin boards. Their classes were large, 
and there were fewer support staff. We had the impression, from our time at Sycamore, that just a 
few more resources and a little more support would have made a huge difference to their overall 
attainment of their students and VA scores of their teachers. 

The ecological dimensions of resources and coherence worked together to set a context for 
quality in each of the school communities we studied. Rather than capturing a characteristic or 
capacity of individual teachers, we argue that the metrics of quality are part of a broader community 
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narrative in which not all participants have equal opportunities. These histories helped us understand 
schools like Hickory and Sycamore that do not fit a simple relationship between VA and classroom 
quality at the school or classroom level. 

This study’s focus on context echoes research by scholars using social network analyses to 
understand the efficacy of school reforms. Rather than assuming that teachers are individual agents 
whose capacity is a matter of skill, will, and knowledge, social network analyses argue that resources 
are activated in social contexts. This makes an individual approach to quality, reflected in VA, 
problematic: 

Rather than presuming that estimating a teacher’s value-added in student 
achievement is the best indicator to use in selecting teachers into the profession, 
a social capital approach views teaching less as a solo practice and more as a 
practice that evolves within a broader social and organizational contexts (Penuel, 

Frank, Sun, & Kim, 2012, p. 198). 

Individual teachers are actors in the creation of educational quality but it is also true that the 
resources in the context such as funding, leadership, and collaboration are actors as well 
(Darling-Hammond et al, 2012). Quality in context is the punch line of this paper. 

Comparing two metrics of quality—VA and CLASS scores—within the context of the 
capacity of schools to enhance that quality, provides a window to the importance of ecologies in 
education. It provides a deeper sense of how teacher or school quality, as measured by VA, is nested 
within classroom or school quality, as measured by CLASS. It also highlights how student outcomes 
are made possible through classroom interaction and processes that are supported or undermined by 
contextual factors. Both types of quality are supported within local contexts and through 
engagements and interactions that make only certain things possible. Without this ecological view of 
quality, VA data alone tells us little about how to improve student achievement. Though the 
calculation of VA purports to control for demographic characteristics, there is more to context than 
student characteristics. A contextual look at quality provides us with a more nuanced understanding 
of the ways that institutions, in addition to individuals, create educational opportunities that translate 
into achievement. 
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Appendix A 

Learning Team Focus Group Interview Date:_ 

Participants:_ Interviewer:_ 

1 Tell me a little bit about your school and the community it serves. 

2 Student achievement is a big focus in education currently. How would you describe this 
school’s student achievement? 

a What is your school doing to address student achievement? 

3 The district has, like many other districts, put resources into enhancing the ability to use data 
at all levels of the system. What is happening at the district level to support data use? 

a How is your school being supported in this effort? 

b What data have been provided and how do you use them? 

4 At this school, what assessments do you use to generate data? 

a How are these assessments related to your curriculum? 

5 There are many ways to measure how a school is doing. One increasingly popular way is 
Value Added. As a Learning Team, how would you describe Value Added? 

a How does the district use VA? 

b How do you use VA in your work supporting school staff? 

6 Policymakers and the public have been calling for classroom measures of quality. Value 
Added is one of the things bandied about as a tool. What other elements do you consider 
when you think about classroom quality? 

a How do you use these elements when you work as instructional leaders, supporting 
teacher development? 

7 Is there anything else I should ask that would be important for me to know? 
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